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ABSTRACT 



In an image object managing technology, a copyright and a 
right of portraits of images such as mobile pictures can be 
easily managed. When a producer creates a stream of video 
images, an identical identifier is assigned in all video 
images, in response to indication from the producer, to 
object streams of objects which is regarded as identical by 
the producer. To retrieve a particular object later from the 
video stream, the particular key assigned in the creation of 
the video stream is employed as a retrieval key. The object 
stream includes an encryption field. Using an encryption key 
only known by the producer, at least part of data of the 
identifier or data field is encrypted to be written in the 
encryption field. An identifier extracted from data obtained 
by decoding the encryption field using the encryption key is 
compared with an identifier in the object stream to thereby 
detect modification of the identifier. 

10 Claims, 13 Drawing Sheets 
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IMAGE OBJECT MANAGING METHOD, AN 
IMAGE PROCESSING APPARATUS USING 

SAID METHOD, AND A RECORDING MEDIA 
FOR PROGRAMS ACHIEVING THE SAME 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to a technology to encoding 
images such as mobile pictures, and in particular, to an 
image object managing method of managing objects con- 
figuring images by use of identifiers, an image processing 
apparatus using the method, an image stream produced by 
the method and a recording media for the stream, and a 
recording media on which programs achieving the method is 
recorded. 

2. Description of the Related Art 

Heretofore, there have been proposed a large number of 
methods of encoding or coding mobile pictures. One of these 
methods which is standardized by a moving picture coding 
experts group (MPEG) has been broadly employed in the 
present stage of art. The MPEG standard specifications 
include MPEG1 and MPEG2 at present. While MPEG1 is 
adopted for video compact disks (CD) and MPEG cameras, 
MPEG2 is utilized for digital video disks (VD) and digital 
satellite broadcasting. As general features of the MPEG 
standards, there is included data compression according to a 
correlation with respect to time, which leads to high coding 
efficiency. In this compression method, differences between 
a plurality of frames (screen images constituting a mobile 
picture) which are continuous with respect to time are 
recorded to compress data. In general, adjacent frames of a 
mobile picture are quite similar in image features to each 
other and hence a high compression ratio can be obtained by 
coding the difference therebetween. This is why the com- 
pression efficiency is improved. 

New specifications of MPEG4 are being prepared or 
discussed at present. This is developed inherently for com- 
munication and is different in features from MPEG1 and 
MPEG2. Description will be, however, given of only sec- 
tions of MPEG4 which relate to the present invention. 
MPEG4 is conspicuously different from MPEG1 and 
MPEG2 in an object encoding operation. According to 
MPEG4, each object appearing on a screen can be encoded. 
Consequently, it is possible that a person (an assassin) 1 and 
a background 2 of one film scene are respectively encoded 
to be also respectively transmitted, and it is possible, on a 
side having received the respectively transmitted signals, to 
combine the images into one scene. In this operation, each 
object is encoded, for example, as follows. Although a little 
change appears between successive scenes in actual mobile 
pictures, a large change takes place between scenes in this 
example for easily understanding in the description of this 
application. 

First, an image of a person is obtained before a single- 
color background in a studio using, for example, a method 
called "blue back". Thereafter, a rectangle in which the 
person's image is completely contained is defined in the 
overall image and is trimmed as shown in FIG. 8. Using as 
a key the single color of the background, a mask is produced 
to separate the person from the background (FIG. 9). FIG. 8 
is compressed by a discrete cosine transform (DCT) which 
is similar to ordinary MPEG2. The mask, i.e., FIG. 9 is 
similarly compressed. Compressed data of FIGS. 8 and 9 is 
transmitted to a partner (on the receiving side). The receiver 
decodes the data of FIGS. 8 and 9 and then recognizes that 
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a white area of FIG. 9 is "transparent" to set the background 
area of the decoded FIG. 8 to "transparent" in accordance 
with the mask of FIG. 9 (reference is to be made to FIG 10). 
Thereafter, the obtained image is combined with a back- 
5 ground image separately prepared. 

This method has a feature of higher encoding efficiency 
when compared with a method in which the overall screen 
is encoded. This is because of a principle that the back- 
ground screen is almost still, i.e., little change, and hence the 
10 quantity of data to be processed is small, namely, only the 
moving sections of the person are to be encoded. 

The present invention provides, in consideration of image 
compression technologies such as MPEG4 having the fea- 
ture above, novel characteristics to achieve an object-base 
15 encoding operation. 

A video image stream of MPEG4 includes a stream of 
backgrounds and streams of respective objects. FIG. 4 
shows an example of MPEG4 streams corresponding to 
FIGS. 1 to 3. 

20 

A stream 10 includes individual object streams 11 to 19 
and a control stream 20 describing a composite rule of these 
object screams (the description stipulates, for example, 
positions of images with respect to depth in the screen image 

25 and timing of appearance of images). 

In FIG. 1, an assassin 1 and a background 2 are related to 
object streams U and 12, respectively. In FIG. 2, a target 3, 
a background 4, and a hindrance 5 are associated with object 
streams 13 to 15, respectively. In FIG. 3, a target 6, a 

30 background 7, an assassin 8, and a hindrance 9 are related to 
object streams 16 to 19, respectively. Object streams 13 and 
16, 15 and 19, and 14 and 17 are not interrupted therebe- 
tween and are hence respectively continuous object streams, 
i.e., each combination forms one object stream. 

35 Stream 10 is subdivided in a time division procedure into 
small packets to be transmitted. Consequently, for the 
receiving side of these object streams (and the control 
stream) to restore the original streams, there are required 
identifiers to identify the respective object streams. Accord- 

4Q ing to the stipulation of MPEG4, only the number of bits is 
determined for the assignment of identifiers and no other 
rules are stipulated for the identifiers. Therefore, for the 
identifiers, serial numbers are ordinarily assigned in an order 
of appearance of objects. To guarantee the time sequence of 

4S the objects, a time stamp is assigned to each object stream. 
Even in a case in which objects which regarded as the 
same object by viewers, for example, objects corresponding 
to an identical person appear in different scenes, if the 
pertinent stream is once interrupted, a subsequent scream is 

50 ordinarily assigned with a new serial number. Namely, 
another number is assigned to the subsequent stream. In 
consequence, when at least the stream is simply analyzed by 
a computer, a correlation, i.e., the person appearing in two 
or more scenes cannot be appropriately identified. 

55 Consequently, the prior art is attended with problems as 
follows. 

An amateur can produce copies of video images only for 
curiosity or entertainment to distribute the copies via a 
network. Moreover, it may also be possible that a malicious 

60 person who aims at disgracing dignity of a particular person 
copies images of the person appearing in video images and 
combines the copied images with another background and 
other objects to produce original video images. There exists 
fear that such an act infringes the right of portraits of the 

65 person. Additionally, the produced screen images are not 
associated with intention of the producers or programs or 
films. Namely, there also exists fear of an infringement of a 
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copyright. Particularly, when the user of the right is to be stipulated to assign an identifier. Therefore, even if the 

charged, there possibly occurs a matter of money. identifier is changed, no problem occurs in the decoding 

When the video images are processed in the analog format phase. Namely, it is possible for the malicious person to 

or in MPEG2 and preceding digital formats, these actions do change the object identifier. In this situation, however, the 

not easily occur in general because of difficulty in separating 5 object of which the identifier is changed cannot be retrieved 

a person from a background thereof. However, since object by the computer. Consequently, there is required a measure 

streams can be separated in accordance with MPEG4 under to detect the act of changing the identifier. This has been 

discussion, the problem above may frequently arises. impossible in the prior art. 

Therefore, a producer who provides video images in a ~ . . MI , - . 

r . . . j- t ■ * j . £ Description will now be given of a case in which a 

format in which the encoding of images is earned out for in , r r ^ , , . 

. . _ . wrk „ A . . * . 4 , . , , 10 producer of programs or a user having a stream desires to 

objects as in MPEG4 is required to manage the copyright * j- ♦ • j c *u * tu- a. * 

J . « • 1 . , . . r, . create a digest or an index of the stream. This may occur that 

more strictly than in the job in which video images are ^ , . . , , / 

« . . — mro J , . - „ * . . the user produces the digest or mdex as he or she likes or a 

encoded in MPEG or any preceding format. For this j- »• j -j * j • * j 

, . . 0 - digest is produced for a program guide or an index is created 

purpose, it is necessary to obtain streams of video images c & * f ^™ j 

■1 li ■ .1. 1 * * j * u *u . for a media title for DVD video images, 

available in the market so as to determine whether or not 1 s & 

particular objects exist therein. In the prior art, only the Id ^ situation, even if it is desired to make a search or 
object identifiers can be used as information to identify to CTeate a database for each person in the images, since 
associated objects. The identifiers are serial numbers identifiers assigned to objects are not associated with such 
assigned in a sequence of appearance of video images and persons, it is not possible to simply search for each person 
are not associated with the contents of the objects. Namely, 20 m a mechanical manner. The identifiers are attended with a 
there is no measure for the computer to easily identify difficulty, in addition to the creation of a database, for an 
objects and hence it is impossible for the computer to operation to select only scenes in which a favorite actor 
retrieve particular objects. In consequence, there exists only appears or to generate an index of such scenes. These jobs 
one available method, namely, the video images are required have been conventionally carried out by humans. However, 
to be visually checked by humans. « ™ me present situation in which the number of video sources 
The situation above will be more specifically described by a f e «maikably increased due to, for example multi- 
referring to FIGS. 1 to 3. ITiese images show a scene of a *b*nnt\* of satellite broadcasting the method above is 
film in which an assassin with a machine gun attacks his 1 " mtod * usability and hence it has been desired to conduct 
target. FIG. 1 shows a close-up image of the assassin. In ^rations by machines. Particularly, when a user desires 
FIG. 2, the attacked person is running away. FIG. 3 shows 30 ? °°u J ° • U ls t nccessar y m ^ ^° vc ^onal method 
the running person and the assassin viewed from his rear for to view ^contents many times to manually 
side. It is assume that a cut takes place between the images mark necessary points. TTiis requires a conspicuously large 
respectively of FIGS. 1 and 2 and the camera pans (moves amount of human labor 

in a horizontal direction) from FIG. 2 to FIG. 3 to resultantly Description will now be given of simplification of video 

show the assassin in the image. In FIG. 1, assassin 1 and 35 images. In a scene, for example, a mob scene (in which 

background 2 are related to respective objects and are many people are moving), when a particular person or object 

therefore encoded separately. In FIG. 2, objects are produced is not easily identified in the scene, it maybe desired to thin 

for target 3, background 4, and hindrance 5, respectively. In out other objects to some extent to attain video images in 

FIG. 2, target 3 runs away and recognizes hindrance 5 to which the particular object is clearly shown. The user may 

consequently change the running direction. In FIG. 3, assas- 40 desire to view the target item concealed by an object existing 

sin 8, target 6, and background 7, and hindrance 5 are in fr° nt of the item or to extract, in a weather forecast, a 

associated with objects. desired region or a desired item (e.g., only the temperature 

Since no cut exists between FIGS. 2 and 3, the same or height of waves). Specification for deletion of a particular 

identifier is assigned to targets 3 and 6 (reference numerals for extraction of a particular object, or for deletion of 

13 and 16 of FIG. 4). This is also the case of hindrances 5 45 aU wWdl conoeal a P^^^ <*i«* * to bc manu " 

and 9 as well as backgrounds 4 and 7 (reference numerals 15 ^ conducted in the prior art. Resultantly, when objects 

and 19 as well as 14 and 17 of FIG. 4). However, a cut exists move frequently or when many cutbacks occur in video 

between FIGS. 1 and 2, assassins 1 and 8 are mutually images, me desired operation is required to be conducted for 

assigned with different identifiers (reference numerals 11 each movement or each cutback. It is therefore actually not 

and 18 of FIG. 4). This is also the case of backgrounds 2 and 50 P 0 ^ 0 ^ to achieve the operation in a realtime fashion. 

4, i.e., there are assigned different identifiers (reference According to the circumstances above, it is not possible in 

numerals 12, 14, and 18 of FIG. 4). For the producer having the prior art to appropriately manage the copyright and the 

the copyright, assassin 1 in a scene is the same as assassin right of portraits for mobile pictures in films, television 

8 in another scene. That is, for an actor of the assassin, these programs, DVD images in a simple method, 

video images are protected in accordance with the right of 55 SUMMARY OF THE INVENTION 
his portraits. In the object-base encoding, mobile pictures of 

objects can be easily separated, namely, it is easy to combine It is therefore an object of the present invention to remove 

assassin 1 with background 7 to create a new image. The the drawbacks of the prior art and to provide an image object 

obtained image, however, does not match intention of the managing method capable of simply achieving management 

film producer. 60 of the copyright and the right of portraits in a sequence of 

Assume now that the drawback above is removed by images such as mobile pictures, an image processing appa- 

some measure for the computer to identify objects. A mali- ratus employing the managing method, image streams cre- 

cious person will deceive or cheat the measure. In accor- ated m the method and a recording media for the image 

dance with the present invention, identifiers of objects are streams, and a recording media on which a program to 

employed as the measure, which will be described later. In 65 implement the method is recorded. 

this case, a malicious person may possibly change an Another object of the present invention is to provide an 

identifier of an object. In MPEG, no particular rule is image object managing method suitable for creating a digest 
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and/or an index of a sequence of images such as mobile of the object stream in the areas other than the encryption 
pictures, an image processing apparatus employing the field is recorded in the encryption field, 
managing method, image streams created in the method and 

a recording media for the image streams, and a recording BRIEF DESCRIPTION OF THE DRAWINGS 

media on which a program to implement the method is 5 ^ ofc(jecls ^ features of the presem invention ^ 
recorded. become more apparent from the consideration of the fol- 

Still another object of the present invention is to provide lowing detailed description taken in conjunction with the 
an image object managing method suitable for extracting a accompanying drawings in which: 
particular object from a sequence of images sucb as mobile pj GS ito3aK showing ^ embodiment of an 

pictures, an image processing apparatus employing the io objec , encoding operation in MPEG4 for respective objects 
managing method, image streams created in the method and m accordance the present invention; 
a recording media for the image streams, and a recording __ , . .. . ... . _ ., . , . A 

media on which a program to implement the method is FIG. 4 is a diagram for explaining in detail of object 
recorded streams in accordance with the present invention; 

^To achieve the objects above in accordance with the 15 . FIG. 5 is a ^ a S^ am explaining in detail of encryption 
present invention, when a producer creates a stream of video m accordancc ^ ^ P rcseDt «™ntion; 
images, it is allowed for the producer to intentionally assign ^G. 6 is a diagram for explaining an example of extrac- 

an identical identifier in all video images to object streams tion of objects in accordance with the present invention; 
of objects which is regarded as identical by the producer. FIG. 7 is a flowchart for explaining an algorithm to extract 

Consequently, when a particular object is desired to be 20 objects in accordance with the present invention; 
detected in the video stream, the identifier assigned to the F]<3. 8 is a diagram for explaining an original image of an 
particular object can be used as a retrieval key. Namely, the object in MPEG4 compression; 

particular object can be retrieved by a computer. FIG. 9 is a diagram for explaining a mask for the object 

Moreover, in accordance with the present invention, there 25 in MPEG4 compression; 
is provided an image object managing method of mobile p, G 10 ^ a diagram f or explaining an image after 
pictures and the like in which respective images appearing processiDg of a background for the object in MPEG4 com- 
in backgrounds and screen images are independently pression* 

encoded such that in a decoding phase, the background and ' . _ , . . , „ 

screen images are decoded to be combined with each other 3Q FIG. 11 is a diagram for explaining control stream 20; 
for presentation thereof. In the method, there is provided a FIG. 12 is a diagram showing an example of tables used 
database which establishes a correspondence between a to extract calculation formulae of objects; 
required condition for a data stream and an identifier of the FIG. 13 is a diagram showing an example of tables 
data stream or a value obtained by conducting a ma them a ti- employed to extract information "region" of objects; 

cal operation for the identifier. When an entire stream is to 35 FIG. 14 is a diagram showing an example of tables 
be created, each object satisfying a desired condition asso- employed to extract information "mark type" of objects; 
ciated with the database is assigned with an identifier FIGS 15 and 16 arc diagrams for explaining a principle 
corresponding to a mathematical operation satisfying a 0 f an electronic watermark; and 

condition of the data stream, the identifier being an identifier m ™ . - * 

of an object stream associated with the object, llie other m nGS 17 and 18 m tLil « rams for "plaining a hierarchy 
object streams are assigned with an identifier other than that streams. 

of the object stream. When extracting from the overall DESCRIPTION OF THE PREFERRED 

stream each object stream satisfying a desired condition EMBODIMENTS 
appearing in one scene or in a plurality of scenes, the 

identifier corresponding to the desired condition is used as a Referring now to the accompanying drawings, description 
key for the retrieval of the object stream. wll be given of embodiments in accordance with the present 

The image object managing method further includes the invention, 
steps of disposing an encryption field in a subordinate field Fiist > me wndamental concept of the present invention 
of the object stream, encrypting all or part of data of the wu 'l ^ described. 

object stream in the areas other than the encryption field 50 MPEG4 provides a new standard to compress mobile 
including the identifier field, using as a seed an encryption pictures and has an object compression function in which 
key known only by a person who encrypts data; and writing respective objects appearing in screen images, for example, 
encrypted data in the encryption field, extracting an identi- persons and cars having various contours can be separately 
fier from data decoded using the encryption field and the compressed to be combined with a background. When data 

encryption key. Alternatively, the method includes the steps 5S is encoded, an identifier is assigned to each object. However, 
of creating an electronic watermark using the identifier as a MPEG4 has no special rules for the assignment of the 
seed, writing the watermark in a data field of the object identifier. In accordance with the present invention, an 
stream, and extracting an identifier from the electronic identifier assigned to an object is regarded as unique thereto, 
watermark buried in the data field. The method further Consequently, in an object retrieving phase, the identifier 

includes the step of comparing the extracted identifier with m can be used as a key to easily identify the object 
an identifier in the object stream, thereby detecting modifi- Next, referring to FIG. 4, description will be given of an 
cation of the identifier. embodiment in accordance with the present invention. 

In accordance with the present invention, there is pro- FIG. 4 shows an abstract image of video image streams 
vided recording media containing a plurality of object corresponding to FIGS. 1 to 3. Stream 10 includes respective 

streams recorded thereon in which each object stream 65 object streams 11 to 19 and control stream 20 which is a 
includes an object identifier field, an encryption field, and a stream including descriptions of combination rules 
data field, and data obtained by encrypting all or part of data (positional relationships) between a plurality of objects and 
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points of output timing in a time axis. This configuration also is subdivided into a data field, an identifier field, an subor- 

applies to general MPEG4 streams. The stream can be dinate field, etc. If the object identifier field of MPEG4 is 

divided into sub-streams corresponding to FIGS. 1 to 3, insufficient for the number of figures, it is possible to use a 

respectively. Each stream proceeds in a time axis, which is field (not reserved) defined as a field for expansion or 

represented by an abscissa, i.e., a horizontal line. In this 5 extension of object attributes in the future. However, when 

stream, object streams U to 19 corresponds respectively to the subordinate field is used to define the identifier, there 

objects 1 to 9 shown in FIGS. 1 to 3. takes place a drawback, for example, an access to the 

An assassin is associated with object streams 11 and 18. subordinate field takes a long period of time. Therefore, if 

These streams are separated from each other and hence it is possible, the identifier stored in the object identifier field 

not guaranteed that these streams are assigned with an 10 desirably includes its own meanings, 

identical identifier. However, these object streams are Description will now be given of a method of retrieving 

related to one person and hence are assigned with an a particular object in accordance with the present invention, 

identical identifier in accordance with the present invention. As can be seen from FIG. 4, control stream 20 extends to 

Due to the identifier, object streams 11 and 18 are interpreted cover all other streams. FIG. 11 shows details of control 

to include an identical person. By applying this rule to the 15 stream 20 including a few data items. Numbers (11) and (12) 

overall images of a film, an identical identifier is assigned to Q n the left-hand side (corresponding to FIG. 1) express that 

the pertinent person. object stream 11 (assassin 1) is placed at the nearest position 

For assassin 1 of FIG. 1 and assassin 8 of FIG. 3, a to viewers and object stream 12 (background 2) is placed 

copyright is possessed by one person and each object is behind object stream 11. Numerals (13), (15), and (14) 

respectively assigned with an identifier. In accordance with 20 similarly means that object stream 13 (target 3) is at the 

the present invention, however, the objects of FIGS. 1 and nearest position to viewers and object stream 15 (hindrance 

3 are assigned with the same identifier. In general, it is 5) and object stream 14 (background) are placed behind the 

difficult to apply the rule to all objects. Therefore, the rule others in this sequence. Numerals (18), (19), (16), and (17) 

is applied to objects which claim a copyright, for example, at a position of last timing (associated with FIG. 3) indicate 

objects of a famous film star. The other objects will be 25 that object stream 18 (assassin 8) is at the nearest position to 

assigned with sequential numbers as in the prior art. In viewers and object stream 19 (hindrance 9), object stream 16 

general, an identifier will be first actually assigned by (target 6), and object stream 17 (background 7) are placed 

persons of an office associated with the film star. Next, behind the other streams in this order, 

description will be given of an example of the method of first ^ jhe contents of control stream 20 are represented in a 

assigning an identifier to objects. quite simple manner, i.e. , identifiers of object streams appear 

For the management in accordance with the present at appropriate timing. Therefore, it is a feature of the control 

invention, there is determined a maximum number of objets stream that the data volume thereof is considerably reduced 

to reserve accordingly numeric values to be assigned as when compared with other object streams including a large 

identifiers of the objects, the number of the objects being 35 amount of mobile picture data. Consequently, when only the 

equal to that of the numeric values. When object streams are control streams are extracted to be stored in a database in a 

first created, identifiers other than the numeric values are storage, it is possible to know points of timing of appearance 

assigned to objects. Thereafter, while visually checking of a desired object in a screen only by conducting an 

object streams, operators manually rewrite the identifier information retrieving operation through the database. Since 

(appropriately allocated in a preceding step) of each desired ^ necessary objects are assigned with an identical identifier in 

object stream, i.e., a desired identifier is written over the accordance with the present invention, it is necessary to 

existing identifier. In this method, the automatically conduct a search only through control streams 20 to retrieve 

assigned identifiers and the manually assigned identifiers are scenes in which a necessary object, e.g., a particular film star 

separately and appropriately allocated to the object streams. (assassin 1 or 8) appears in the example above. 

It is possible to suppress a disadvantage in which an iden- 4S Referring now to FIG. 6, description will be given of an 

tifier akeady assigned is allocated to another stream appear- example in which objects having a specified meaning are 

ing in one screen. There may be used an image recognition extracted from a large number of objects by use of meanings 

technology to evaluate a feature and a degree of similarity of identifiers so as to thereafter display the extracted objects, 

between images to automatically assign identifiers to nG 6 M e k of a map m a weather 

objects. Even in such a case, the computer need only 5Q forecast ^ example mc i u des a complex weather map 26 

evaluate and determine each desired object stream, and the and a ^ le 2? Wheo a fe 

other necessary operations can be achieved in accordance ^ ^ accordance ^ Mp EG4> me map fe repre . 

with the algorithm of the present invention. Furthermore, in ^ b data ^ ^ object of a background 

accordance with a recognition ratio .of the image recognition representing a map aad obje cts, for example, of weather 

technology, the automatic identifier assignment through 55 marks (clear? ^ doud etc )> values of tempera ture 

image recognition may be combined with the manual iden- (maximum ^ minimum values), and height of waves for 

Ufier specification of operators ^such that resultsof the image cach rc ^ on of mc Whcn ^ of ^ itcms m 

recognition are confirmed by humans. displayed in an" ordinary weather forecast, the screen image 

FIG. 4 shows ma simple diagram sequences of objects in G f complex weather map 26 becomes complicated and 

respective streams. Stream 10 includes streams 11 to 19 of 60 cannot be easily understood. Consequently, weather infor- 

various objects. The respective streams are appropriately mation is respectively broadcast in a time axis, for example, 

combined with each other for representation thereof in a omy me weather is first broadcast and then only the values 

time axis. There is also provided control stream 20 to 0 f temperature are transmitted. However, ordinarily, viewers 

supervise the combining procedure (e.g., positional relation- do not require the weather of the all regions of Japan, 

ships with respect to depth of the screen image). 65 name ly, viewers want to generally know weather informa- 

Basically, an identifier is actually stored in an object tion of a region in which they five or regions to which they 

identifier field stipulated by MPEG4. An object stream area are going. To obtain such information in the conventional 
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methods, viewers are required to watch the entire weather the user can extract desired objects from screen images, 

forecast and hence there is required a long period of time. information transmitted in a time division scheme in the 

In accordance with the present invention, by uniformly P rior art for simplification of screen images can be sent at a 

assigning an identifier to each region, it is possible to attain nme - ™s makes it possible to save time, 

all information items of a particular region. In simple 5 Finally, description will be given of an embodiment of a 

weather map 27, the weather and temperature are displayed method of preventing an event in which a third person with 

only for the Kanto region. Namely, only the information a malicious intention illegally changes an identifier. When 

items of Kanto are extracted for the display thereof in such a third person attempts to extract a particular object 

accordance with a particular rule applied to an identifier of stream for an illegal purpose, for example, to plagiarize the 

objects. Specifically, a particular number, e.g., a remainder 10 stream, it is technically easy for the third person to simply 

obtained by dividing an identifier by seven is associated with pass the check above by changing only the identifier. To 

the identifier such that Hokkaido is associated with identi- prevent such an action, a technology of encryption is 

fiers having a remainder of 1 and Tohoku is associated with adopted in this embodiment, namely, there is provided a 

identifiers having a remainder of 2, and so forth. In this mechanism to check the changing of the identifier in this 

fashion, Kanto is related to identifiers with a remainder of 3. 15 example. 

In simple weather map 27 of FIG. 6, the items retrieved in An object stream area includes a plurality of fields which 

association of the identifiers with a remainder of 3 are are generally classified into data, identifier, and subordinate 

displayed. fields. In the subordinate field is written, for example, an 

It may also be possible to establish a correspondence attribute of the pertinent stream, which does not directly 

between a quotient and a type of weather information. For 20 exert influence on the decoding of data. Information written 

example, identifiers having a quotient of 0 in the method in this field is used later, for example, by the user to further 

above are assigned to whether marks, identifiers having a process objects by a computer. For example, indication of a 

quotient of 1 are assigned to temperature, and so forth. copyright is stored in the subordinate field. In general, are 

Resultantly, it is possibly to display only the weather infor- served field is provided. In accordance with MPEG4, it has 

mation or only the temperature values. The quotient and the 25 already been determined to define a reserved field for use in 

remainder may be combined with each other to extract the future. 

desired information items only for the Kanto region. If these FIG. 5 shows a format of an object stream used in this 

operation modes are selected by the user, the user can embodiment. The format includes an identifier filed 21 to 

immediately obtain desired information at a glance. When it identity the object stream, an encryption field 22, a subor- 

is desired to extract information for a combination of 30 dinate field 23, and a data field 24. A person who encrypts 

"Kanto" and "whether and temperature", the user need only data selects all of or part of subordinate field 23, data field 

select a combination of identifiers "3" and "4" for which the 24, and identifier field 21 and encrypts the selected items 

remainder is 3 and the quotient is 0 or 1. using a secret encryption key 25 known only by himself or 

The procedure will now be more generally described by 3S herself and stores or buries the resultant data in encryption 

referring to a flowchart. field 22. 

FIG. 7 shows in a flowchart an algorithm to achieve the When attempting an illegal action for data in the object 

method above. For identifier n, there exist a plurality of stream format, it is necessary for a malicious third person to 

calculation formulae depending on types of desired infor- change the contents of identifier field 21 so as to thereafter 

mation to be extracted. Assume that one variable x is ^ write an appropriate encryption in encryption field 22. 

assigned to each type of information. Then, two kinds of However, even if data of subordinate field 23 and data field 

expressions exists for f{x,n) for a region (x-1) and weather 24 is known, it is technologically quite difficult to obtain the 

(x=2).FIG. 12 shows a correspondence in this case. Namely, encryption key in encryption key field 25 using the data. 

f(l,n) Mod(7,n) represents a remainder of n/7 and f(2,n)= Namely, an appropriate encryption cannot be produced on 

Quotient(7,n) designates an integer part of the quotient of 45 the basis of the new identifier in identifier field 21. 

n/7. Information toe be extracted for x=l is "region", which When checking any illegal action for the encryption, the 

is specifically shown in an example of FIG. 13. Information person who encrypts data generates encrypted data using 

desired for x=2 is "mark type", which is concretely shown data of identifier field 21, subordinate field 23, data field 24, 

in an example of FIG. 14. and encryption key field 25 to compare the obtained data 

To extract only desired information, the user first attains 50 with that stored in encryption field 22. If these items match 

x according to the kind of information to be extracted (step each other, it can be assumed that there exists no illegal 

101) and then a solution y from a table corresponding to x action. Otherwise, it is assumed that an illegal action has 

(step 102). For each n, f(x,n) is calculated (steps 103 and been achieved. Using this operation, the encryption gener- 

104) to determine whether or not fifcn) is equal to y (step aung person who strictly manages the copyright can make a 

105) , and only if fi%n)=y, n is extracted (step 106). In the 55 check to determine presence or absence of an illegally 
example of FIG. 13, the remainder of f(l,n)-Mod(7,n) created object stream. On the side of the copyright holder, an 
indicates a region, specifically, the remainder of " 1", "2", identifier not generally used, e.g., a large value of 255 if the 
"3", "4", "5", "6", and "0." denote Hokkaido, Tohoku, Kant identifier includes eight bits, is assigned to an object asso- 
o, Chubu, Chugoku, Shikoku, and Kyushu, respectively. In ciated with a copyright. 

the example of FIG. 14, the quotient of fi(2,n)=»Quouent(7,n) 60 In this method, however, when the encryption field itself 

designates weather information, concretely, the quotient of is deleted, information of the encrypted identifier in identi- 

"0", "1", and "2" indicate the weather mark, the fier field 21 becomes also unknown. That is, the method is 

temperature, and the wave mark. incomplete that the deleted state cannot be discriminated 

Thanks to provision above, the user can extract a desired from a state not encrypted, 

object of a plurality of desired objects from a large number 65 A plurality of object streams shown in FIG. 4 are par- 

of mobile picture objects; moreover, the user can select only ticularly useful for satellite digital broadcast. However, the 

desired information from complicated screen images. Since object streams may be recorded on a recording media such 
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as a DVD or a CD-ROM to be put to the market. In such a ducted for particular points 41 to 44, 51 to 54, and 61 to 64, 

case, each object stream includes object identifier field 21, four-bit information can be buried in the image. To bury 

encryption field 22, data field 24, etc. as shown in FIG. 5. All more information in the image, the number of particular 

or part of the object stream data other than that of encryption points is increased or the processing is conducted for a 

field 22 (such as values of the identifier and data fields) is 5 plurality of frames to disperse information with respect to 

encrypted to be stored in encryption field. Thanks to this time. 

operation, when the identifier is altered, the action of alter- To extract the watermark, it is only necessary to invert the 

ation can be immediately detected and hence any illegal use processing above. That is, an image is first obtained, data of 

of a copyright can be examined. th c particular points are read therefrom, and a check is made 

Description will now be given of a method in which an 10 for a color difference between upper-left point 71 and the 

electronic watermark is adopted in relation to the encryp- other points. If there are found three of more horizontal 

tion. points (e.g., points 31 to 34) in which "two or more" is added 

Utilizing identifier field 21 and encryption key 25 of FIG. to me value of each P° mt > li & assumed that information " 1" 

5, an digital watermark is created for data field 24 in has be&n inserted therein. If there are found three of more 

accordance with a digital watermarking technology. Namely, 15 horizontal points (e.g., points 31 to 34) in which "two or 

the data is modified and an encryption is inserted therein more " * subtracted from the value of each point, it is 

such that the data field containing resultant data, e.g., data of assumed that information "0" has been inserted therein. This 

images and voice is not strictly equal to the original data. operation is repeatedly achieved for the respective horizon- 

However, when the resultant data is decoded and is viewed tol Lncs and for a plurality of frames if necessary, 

or is heard by a human, it is almost impossible to detect the 20 In accordance with the principle above, data can be buried 

difference between the recorded data and the original data. in image data and the buried data can be read therefrom. 

In general, there is employed a method of spread spectrum. When an identifier is inserted in this method, data can be 

In the method using the digital watermark, identifier field 21 buried without using an extended field. Even if part of the 

can be restored from data field 24 and encryption key field image is slightly altered, the encryption cannot be com- 

25 by using data field 24. The restored data is compared with 25 pletely erased. Consequently, the encryption field can be 

the current identifier. If these items match each other, it can extracted from the remaining image to thereby read the 

be assumed that the identifier is kept unchanged. In this identifier data therefrom. By comparing the obtained data 

method, information of the encrypted data in identifier field with that stored in the identifier field, whether or not the data 

21 can be deleted from the data stream only with the key of has been modified can be determined. If the watermark is not 

encryption key field 25, ensuring safety of the system. 30 originally provided, "information missing" is determined. 

Description will be given in detail of a method of encryp- It is to be appreciated that any one of various watermark- 

tion using an digital watermark. ing technologies may be used without deteriorating the 

The watermarking encryption is an encryption method in advantageous effect of the present invention, 

which data of images and voice in the data field is modified 35 Description will now be given of a hierarchy of streams, 

such that the change is not perceivable for humans and Streams can be configured in a hierarchy in accordance 

information of the data field is stored as encrypted data. with the standard of MPEG4 currently under discussion. 

FIGS. 15 and 16 are diagrams for explaining the principle of FIGS. 17 and 18 schematically show a hierarchic layout of 

the watermark. As can be seen from FIG. 15, a plurality of streams. As can be seen from FIG. 17, a stream 80 includes 

particular position are set in a screen area 30 to determine an 40 streams 82 and 84. Streams 82 and 84 include identifier and 

image area of an appropriate size at each position. This extension fields 81 and 83, respectively. Stream 82 includes 

example includes particular points 31 to 34, 41 to 44, 51 to object streams 86 and 88, which include identifier and 

54, and 61 to 64. Each particular point is assumed to be extension fields 85 and 87, respectively. Stream 84 includes 

represented by a 2-bits by 2-bits area. FIG. 16 shows a object streams 90 and 92, which include identifier and 

layout of the particular point. 45 extension fields 89 and 91, respectively. When the streams 

In general, in an overall image area, adjacent points are ire configured in a hierarchic structure, objects can be 

characteristically of similar colors. In FIG. 16, therefore, it classified into groups. The hierarchic structure facilitates 

is to be assumed that point 71 of an upper-left area is almost operations to edit and to retrieve objects in a screen image 

the same in color with other points 72 to 74. To insert or to including many objects. 

bury information in this image, the color of only point 71 is 50 If the present invention is applied to streams 82 and 84, 

slightly altered. For example, to bury information "1", two when a new stream 93 is created by subdividing streams, 

is added to the value of each of red (R), green (G), and blue identifier and extension fields 81 and 83 attached to the 

(B). To bury information "0", two is subtracted from the streams are deleted regardless of use of the extension field 

value. In the operation, if the value to be added or subtracted or the digital watermark. Namely, the advantageous effect of 

is too small, the buried information cannot be discriminated 55 the present invention is removed. To prevent such a 

from noise. If too large, the point becomes conspicuous. disadvantage, it is assumed in accordance with the present 

Therefore, the range of modification is required to be invention that when objects are configured in a hierarchic 

determined in consideration of these conditions. structure, a lower-most hierarchy is encoded, i.e., object 

When the processing is executed only for a single par- streams 86, 88, 90, and 92 are encoded in this example, 

ticular point, the resultant value may appear by chance. 60 Since there exists no fear of deleting the identifier and 

Namely, whether the value is an accidental value or is extension fields, the present invention is particularly effec- 

resultant from the processing above cannot be determined. five. 

Consequently, the processing is achieved for a plurality of As above, the copyright and the right of portraits can be 
particular points. For example, all particular points 31 to 34 easily managed in accordance with the present invention, 
of FIG. 15 are processed for this purpose. It is assumed that 65 The present invention also provides durability against modi- 
values of these points, i.e., four points represent one-bit fica lions. Objects having a particular meaning can be simply 
information. Consequently, when the processing is con- extracted. In accordance with the present invention, MPEG4 
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currently being discussed can be accepted without any 
conflict, it is hence not required to expand or to change the 
MPEG4 standards. 

Although description has been given of embodiments for 
mobile pictures in the MPEG4 format, it is obvious that the 5 
present invention is applicable to any image or video sys- 
tems in which objects of images are managed according to 
identifiers. 

The present invention includes primary sections which are 
supplied in the form of software programs for information 10 
processing apparatuses such as personal computers and/or 
for apparatuses dedicated to image processing, e.g., non- 
linear editing apparatuses. The programs are stored on 
recording media such as a DC-ROM and are installed 
therefrom into the apparatuses above. The programs may be 15 
supplied from a server computer via a network to be 
down-loaded or installed in the apparatuses. 

While the present invention has been described with 
reference to the particular illustrative embodiments, it is not 2Q 
to be restricted by those embodiments but only by the 
appended claims. It is to be appreciated that those skilled in 
the art can change or modify the embodiments without 
departing from the scope and spirit of the present invention. 

What is claimed is: ^ 

1. An image object managing method for use with an 
image processing system in which objects constituting 
image frames are respectively encoded to generate object 
streams and the encoded object streams are decoded in a 
decoding phase to combine the objects for presentation 3Q 
thereof, comprising the steps of: 

displaying a screen for assigning an identifier to an object 
of a first object stream constituting an image stream on 
a display; 

assigning, in response to specification from an operator, 35 
the identifier common to an object of a second object 
stream to the object of the first object stream displayed 
on the display; 

disposing a database establishing a correspondence 
between desired conditions for an object stream and an 40 
identifier of the object stream or a value obtained by 
conducting a fixed mathematical operation for the 
identifier; 

assigning, when creating an overall stream, to an object 
having a desired condition corresponding to the data- 45 
base an identifier corresponding to a mathematical 
operation satisfying the condition of the database; 

assigning identifiers other than the identifier to other 
object streams; and retrieving, when extracting from 5Q 
the overall stream an object stream satisfying a desired 
condition and appearing in one or more scenes, the 
object streams by setting the identifier corresponding to 
the desired condition as a retrieval key. 

2. An image object managing method in accordance with $$ 
claim 1, further comprising the steps of: 

reserving a predetermined number of identifiers specified 

by the operator; 
assigning, when first creating the object stream, identifiers 

other than the reserved ones to respective objects of the 60 

object stream; and 
assigning, in response to specification from the operator, 

again either one of the reserved identifiers to an object 

specified by the operator. 
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3. An image object managing method in accordance with 
claim 1, further comprising the step of interactively issuing 
a request of specification from the operator, via an input 
device, to the object stream presented on the display. 

4. An image object managing method in accordance with 
claim 1, further comprising the steps of: 

making the operator in the specification from the operator 

to specify an object for specification and an identifier to 

be assigned to the object; 
extracting the specified object from an object stream 

constituting an image stream in accordance with a 

degree of similarity of images; and 
assigning an associated identifier to the extracted object. 

5. An image object managing method in accordance with 
claim 1, further comprising the steps of: 

extracting data related to the specified identifier from a 
control stream in an image stream; and 

retrieving, with a key set as an identifier, a desired object 
using the extracted data. 

6. An image object managing method in accordance with 
claim 1, further comprising the steps of: 

disposing an encryption field in a subordinate field of the 
object stream; 

encrypting all or part of data of the object stream in the 
areas including an identifier field other than the encryp- 
tion field, using as a seed an encryption key known only 
by a person who encrypts data; 

writing encrypted data in the encryption field; 

extracting an identifier from data decoded using the 
encryption field and the encryption key; and 

comparing the extracted identifier with an identifier in the 
object stream, thereby detecting modification of the 
identifier. 

7. An image object managing method in accordance with 
claim 1, further comprising the steps of: 

creating a digital watermark using the identifier as a seed 
and writing the watermark in a data field of the object 
stream; 

extracting an identifier from the digital watermark buried 

in the data field; 
comparing the extracted identifier with an identifier in the 

object stream, thereby detecting modification of the 

identifier. 

8. An image object managing method in accordance with 
claim 1, wherein the displaying step displays the object 
streams to be assigned with the identifiers on the display. 

9. An image object managing method in accordance with 
claim 1, wherein one image frame does not include the 
object of the first object stream and the object of the second 
object stream at the same time. 

10. An image object managing method in accordance with 
claim 1, further comprising the steps of: 

matching the identifier with a requisite of the object 
stream; and 

wherein when generating the image stream, assigning the 
identifier to the object streams representing the object 
satisfying the requisite and assigning other identifiers to 
other object streams. 

+ ♦ ♦ * * 
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